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Abstract 

Background: Comparative effectiveness studies using Medicare claims data are vulnerable to treatment selection 
biases and supplemental data ffom a sample of patients has been recommended for examining the magnitude of 
this bias. Previous research using nationwide Medicare claims data has typically relied on the Medicare Current 
Beneficiary Survey (MCBS) for supplemental data. Because many important clinical variables for our specific research 
question are not available in the MCBS, we collected medical record data from a subsample of patients to assess 
the validity of assumptions and to aid in the interpretation of our estimates. This paper seeks to describe and 
document the process used to collect and validate this supplemental information. 

Methods: Medicare claims data files for all patients with fee-for-service Medicare benefits who had an acute 
myocardial infarction (AMI) in 2007 or 2008 were obtained. Medical records were obtained and abstracted for a 
stratified subsample of 1,601 of these patients, using strata defined by claims-based measures of physician prescribing 
practices and drug treatment combinations. The abstraction tool was developed collaboratively by study clinicians and 
researchers, leveraging important elements from previously validated tools. 

Results: Records for 2,707 AMI patients were requested from the admitting hospitals and 1,751 were received for an 
overall response rate of 65%; 1,601 cases were abstracted by trained personnel at a contracted firm. Data were 
collected with overall 96% inter-abstractor agreement across all variables. Some non-response bias was detected at the 
patient and facility level. 

Conclusion: Although Medicare claims data are a potentially powerful resource for conducting comparative 
effectiveness analyses, observational databases are vulnerable to treatment selection biases. This study demonstrates 
that it is feasible to abstract medical records for Medicare patients nationwide and collect high quality data, to design 
the sampling purposively to address specific research questions, and to more thoroughly evaluate the appropriateness 
of care delivered to AMI patients. 

Keywords: Acute myocardial infarction. Medical record abstraction. Medicare, Cardiovascular medications 



* Correspondence: elizabeth-cook@uiowa.edu 

^University of Iowa College of Pharmacy, Iowa City, lA, USA 

Full list of author information is available at the end of the article 

O© 2014 Cook et a!.; licensee BioMed Central Ltd. This is an Open Access article distributed under the terms of the Creative 
BIoIVIGCI Central commons Attribution License (http://creativecommons.Org/licenses/by/2.0), which permits unrestricted use, distribution, and 
reproduction in any medium, provided the original work is properly credited. The Creative Commons Public Domain 
Dedication waiver (http://creativecommons.0rg/publicdomain/zero/l.O/) applies to the data made available in this article, 
unless otherwise stated. 



Cook et al. BMC Health Services Research 2014, 14:391 
http://www.bionnedcentral.conn/1472-6963/14/391 



Page 2 of 1 2 



Background 

There is uncertainty about the best combinations of 
pharmacotherapy for older patients who have experienced 
an acute myocardial infarction (AMI). To examine the 
comparative effectiveness of treatments, an examination of 
the benefits and harms in real world treatment settings is 
essential [1]. Our larger research objective was to assess the 
comparative effectiveness of alternative medication treat- 
ment combinations after AMI on outcomes such as cardio- 
vascular event-free survival, major side-effect risks, and 
Medicare costs. We will be estimating the effectiveness of 
medication combinations by exploiting the real world treat- 
ment variation found in Medicare claims databases for pa- 
tients after an initial AMI. Alternative estimators are 
available to exploit this variation including risk adjustment 
(RA) and instrumental variable (IV) approaches; however, 
the properties of the estimates produced are conditional on 
assumptions specific to each estimator. Three assumptions 
must be valid to successfully use instrumental variables: 
first, the instrumental variable must be relevant and associ- 
ated with exposure; second, that the variable must affect 
the outcome only through the likelihood of exposure; and 
(3) the instrument is unrelated to confounding variables. 
Under these assumptions, IV estimators yield consistent es- 
timates of treatment effects [2]. Alternatively, observational 
data can be used to estimate associations of exposures with 
outcomes, but if exposures are not allocated randomly 
these associations may be confounded by certain character- 
istics such as age, severity of illness, or patient frailty. Thus, 
RA estimators yield unbiased estimates only under this as- 
sumption - that unmeasured factors related to treatment 
choice are unrelated to outcomes - which is difficult to 
prove [2]. The positive properties of both estimators rely 
on characteristics of information by definition unmeasured 
in observational studies. For example. Medicare claims data 
lack important clinical information about the patient health 
status, AMI severity, co-existing conditions, and treatment 
contraindications and complications. As a result, these esti- 
mates are vulnerable to treatment selection bias, which 
arises when physicians tend to treat patients they believe 
will benefit greatly from pharmacotherapy, and tend not to 
prescribe medication to those patients where the risks out- 
weigh potential benefits of treatment. 

To address this concern, we selected a subpopulation of 
Medicare AMI patients from our analysis sample and ob- 
tained medical records to extract information unmeasured 
in Medicare claims for these patients. When conducting 
retrospective studies, in comparison to prospective clinical 
trials, medical record data are considered the "gold stand- 
ard" source for relevant and objective clinical information, 
especially when assessing the effectiveness of medications 
on outcomes [3]. 

This paper describes the process and protocol used 
to collect this supplemental clinical information. We 



employed a complex sampling strategy, collected exten- 
sive data, and relied on hospitals to provide copies of 
the requested records. Detailed study protocols were de- 
veloped and adhered to during the course of this pro- 
ject. The aims of this paper are to describe: 1) the 
permissions required to carry out the study on the 
Medicare population; 2) the sampUng strategy employed 
to identify records for abstraction; 3) the development 
and testing of the data collection tool; 4) the data ab- 
straction process and quality assurance, and 5) plans for 
use of the resulting data. 

Methods 

Faculty and staff from the University of Iowa (UI) College 
of Pharmacy and the College of Public Health, General 
Dynamics Information Technology (GDIT), and Informa- 
tion Collection Enterprises (ICE) worked collaboratively 
on this study, entitled "Comparative Effectiveness of Car- 
diovascular Treatment Combinations Post MI in the 
Elderly". 

Approvals and protections for obtaining medical records 

The study was approved by the Institutional Review Board 
(01-Biomedical) at University of Iowa (#200908724) in 
March 2010. Following approval. Medicare claims data 
were requested from the Centers for Medicare & Medic- 
aid Services (CMS) following the standard process facili- 
tated by the CMS -contracted Research Data Assistance 
Center (ResDAC). Medicare data used by academic re- 
searchers is typically not identified using beneficiary 
names. Social Security Numbers (SSN) or the Medicare 
Health Insurance Claim (HIC) numbers. However, this in- 
formation was necessary for the subsample of patients for 
whom we desired medical records in order to request the 
intended records from hospitals. 

CMS granted a data use agreement (DUA) for Medicare 
enrollment and claims data, as well as permission for the 
collection of medical record data for the study. To ensure 
that we were able to meet our goal of 1,600 abstracted pa- 
tient records (determined by the subsampling methodology 
explained later in this document) we selected a total of 
3,072 patients. To remain compliant with The Health 
Insurance Portability and Accountability Act of 1996 
(HIPAA) Privacy Rule (46 CFR Part 160; Part 164 (subparts 
a,e) [4], CMS approved a process in which ICE acted as an 
"honest broker", functioning as an entity which received 
private identifiers from CMS only, unlinked to claims infor- 
mation. To identify the abstraction subsample, we analyzed 
the de-identified Medicare claims data, selected the cohort, 
and appended the hospital information necessary to request 
records - including the address and phone number of the 
facility as well as admission and discharge dates. A finder 
file of these AMI patients was sent to the Chronic Condi- 
tion Data Warehouse (CCW; www.ccwdata.org) to have 
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identifiers appended, and the file was then sent to ICE who 
obtained and abstracted the records without knowledge of 
the selection algorithm. At the conclusion of the abstrac- 
tion effort, ICE shared with us the abstracted variables ne- 
cessary to conduct our work, but none of the private 
identifiers. Paper copies of the records were shredded at the 
end of the project; however, the records were scanned and 
this information will be available until the DUA expires in 
the event that any data verification is needed. Once the 
DUA expires, the scanned documents will be destroyed. 

Sampling 

We obtained national Medicare enrollment, claims, and 
Part D prescription drug event data files from the CCW 
for patients hospitaUzed with an AMI in 2007 or 2008. 
We used the standard CCW AMI definition for AMI: 
an inpatient stay with the primary discharge diagnosis 
code 410.;vl at any time during the year. The acute hos- 
pital admission date for each AMI served as the index 
date for the AMI. The index AMI stay included all 
Medicare acute and critical access hospital (CAH) 
claims with overlapping admission and discharge dates 
following the initial acute hospital AMI admission. We 
did not consider a patient transferring from one acute 
or CAH to another to be a separate hospital stay. 

We excluded AMIs from our analysis if any of the fol- 
lowing conditions were present: 1) the patient did not sur- 
vive the AMI institutional stay or died within 30 days post 
discharge; 2) the patient had an AMI in the 12 months 
prior to the index admission date; 3) the patient was less 
than 66 years old at the index date to ensure at least one 
year of Medicare eligibility prior to the index date; 4) the 
patient was not enrolled in Medicare Part A and Part B 
fee-for-service during the 12 months prior to the index 
admission date and the 12 months after the institutional 
stay discharge date; and 5) the patient was not enrolled in 
Medicare Part D during the six months prior and the 
12 months after the institutional stay discharge date. We 
further limited the sample by excluding patients who had 
a Medicare hospice claim during the 12 months prior to 
the index admission date, and to ensure that we could 
measure Part D outpatient claims accurately, we required 
patients to survive 30 days post institutional discharge 
date without a readmission to a hospital, hospice, or 
skilled nursing facility [5]. Finally, because we intended to 
examine area-based treatment rates that were based on 
driving times, we eliminated patients who resided in 
Alaska or Hawaii. Our final cohort size was 132,682 post- 
AMI patients. This cohort served as the source population 
for abstraction subsampling. 

We selected our subsample in a manner to enable us to 
assess the assumptions underlying RA and IV estimators. 
We created two key sampling stratification variables: 



1) Observed treatment for each of the three study 
drugs (i.e., statins, beta-blockers, and ACE/ ARBs). 
Each patient was assigned one of eight mutually 
exclusive treatment groups based on observed 
Medicare Part D prescription fills during the first 
30 days post-discharge: a) No treatment; b) statin 
alone; c) beta-blocker alone; d) ACE/ARB alone; 
e) statin + beta-blocker; f ) statin + ACE/ARB; 

g) beta-blocker + ACE/ARB; or h) statin + beta- 
blocker + ACE/ARB. 

2) Area-based treatment ratios (ATRs), a measure of 
local area prescribing practices. We measured these 
ATRs at the patient ZIP code-level using the Driving 
Area of Clinical Care (DACC) method, which re- 
quired that we calculate driving times and distances 
between all ZIP codes in the continental U.S [6]. For 
this analysis, ATRs were determined once a thresh- 
old of 200 patients was met in each geographic area 
[7]. Specifically, we calculated the number of pa- 
tients with AMI within each ZIP code and if there 
were fewer than 200, expanded our range to include 
the next closest ZIP code (using driving time as a 
measure of proximity) and added those people to 
our area. This process was repeated until we con- 
structed treatment areas with least 200 people each. 
The numerator of the ratio was the expected num- 
ber of people treated in that area, based on the char- 
acteristics of the patient and the area (i.e., the 
covariates), and the denominator was the actual 
number of people treated. Once the applicable ATRs 
had been assigned to each patient based on their ZIP 
code of residence, we created flags to identify "high- 
use" (those whose ATR values were in the highest 
quintile) areas corresponding to our eight treatment 
groups above: a) High "no treatment" area; b) High 
"statin alone"; c) High "beta-blocker alone"; d) High 
"ACE/ARB alone"; e) High "statin + beta-blocker"; f ) 
High "statin + ACE/ARB"; g) High "beta-blocker + 
ACE/ARB"; or h) High "statin + beta-blocker + 
ACE/ARB". Of the 132,682 patients, 126,236 (95%) 
fell into one of these high-use ATRs, and became eli- 
gible for subsampling. We chose these stratification 
variables because our resources were limited, and we 
wanted to obtain patients from areas that offered the 
most significant contrast in prescribing rates. If we 
found unmeasured confounders to be similar across 
these areas, this would provide strong evidence that 
the distribution of unmeasured confounders was also 
similar for the areas with rates in-between. 

In addition to the two stratification variables, we 
employed a strategy to ensure geographic balance, to 
reduce the possibility that our comparisons would 
reflect idiosyncratic characteristics of local areas. To 
accomplish this we specified that one-fourth of each 
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stratum for the ATR groups be drawn from each of 
the four U.S. Census Geographical Regions: Northeast, 
Midwest, South, and West [8]. A power calculation 
indicated that a sample size of 200 patients within 
each of the eight ATR groups (for a total of 1,600) 
would provide 80% power to detect a difference in 
mean outcome of 0.36 standard deviation (SD), 85% 
power to detect a 0.39 SD difference, 90% power to 
detect a 0.42 SD difference, and 95% power to detect 
a 0.47 SD difference. Therefore, we designed our 
project budget to allow us to abstract 1,600 medical 
records. We created 64 primary sampling units (PSUs) 
by crossing the eight observed treatments by the eight 
ATR high-use groups, and drew a stratified random 
sample, ensuring balance across the four geographic 
quadrants. A total of 25 abstracted records were 
desired in each of the 64 PSUs. The sampling design 
is shown in Table 1. 

In anticipation of possible non-response to our ini- 
tial requests (1,920 cases: 30 per PSU) for medical 
records, we added a 20% (1,152 cases) oversample to 
our initial wave of medical record requests [9,10]. 
Each sampled AMI stay was given a unique study 
identification number, to allow for de-identification 
of the resulting data. Finally, to ensure that this sub- 
sample reflected the larger cohort from which it was 
drawn, we confirmed that the populations were 
comparable across key variables such as age, sex, 
race and length of stay (LOS). 

Data collection tool 

We created a structured data abstraction tool to obtain 
information from the medical records of the sampled pa- 
tients for the index hospital stay, which could have in- 
cluded treatment at two facilities if the patient was 
transferred during the acute stay. The data collection 
tool was based loosely on an instrument used for a pre- 
vious AMI study called the Cooperative Cardiovascular 



Project (CCP) [11]. We also leveraged key questions 
from other well-designed medical record abstraction 
tools such as one used for the Women's Health Initiative 
[12] and the Adult Comorbidity Evaluation (ACE-27) 
[13]. We examined clinical practice guidelines for AMI 
patients, such as the American College of Cardiology 
and American Heart Association (ACC/AHA) recom- 
mendations [14] and included variables to capture im- 
portant clinical assessment and treatment information. 
Variables were modified and customized in consultation 
with study team cardiologists, internists, and nurses. 
The domains of information and examples of the types 
of data elements captured by the medical record abstrac- 
tion tool are described in Table 2; an exhaustive list of 
all data elements has not been included due to the ex- 
cessive number of variables and descriptions. Oper- 
ational definitions were documented for each variable, 
including a list of valid sources within a medical record 
(e.g., admission face sheet, surgical report, medication 
administration record), inclusion/exclusion criteria, time 
frame parameters, and medical terms/synonyms. To fa- 
cilitate the collection of medications on arrival, we 
imported a list of commonly prescribed medications and 
their corresponding dosages so that a drop-down list 
could be offered to facilitate efficiency; the field also 
allowed for the entry of free text so that medications 
which did not appear on the drop- down list could be 
captured. 

We anticipated that we might occasionally receive a 
medical record for a patient who did not meet our study 
intent (i.e., a patient who was ineligible for secondary 
prevention with drug therapy), so we included some 
"stop abstraction" variables in the data collection tool. 
Incoming records were screened to identify cases dis- 
qualified from a full medical record abstraction, which 
included any of the following: 1) Medical record did not 
indicate AMI was present on arrival - it occurred at 
some point later during the hospital stay, 2) AMI was 



Table 1 Sampling design: stratification by observed treatment combinations and area treatment rates 
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1,600 


S** = statin only; ^B^ = beta-blocker only; ^^A = 
beta-blocker + ACE/ARB. 


ACE/ARB only; SB^ 


= statin + beta-blocker; S^A 


= statin + ACE/ARB; ^BA = beta-blocker + ACE/ARB; SBA = 


statin + 
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Table 2 Description of data elements collection in 
abstraction tool 

Demographic information 

• Diagnosis at admission (STEM!, NSTEMI, unstable angina), arrival 
setting (home or facility, etc.), primary language 

Presenting symptoms, initial vital signs and tests 

• Presentation of chest pain, cognitive status 

• Vitals at admission - pulse, systolic blood pressure, diastolic blood 
pressure. Respiration, temperature, weight and height 

• Lab tests - LDL, HDL, HAlc, ALT, AST, Serum creatinine, cardiac 
troponin level, CK-MB, LVEF, ProBNPXBNP 

• ECG findings (left/right ventricular hypertrophy, complete bundle 
branch block, heart block, AF, other arrhythmia) and Chest x-ray 
findings 

Prior history - conditions, procedures, hospitalizations, and status 
immediately prior to AMI admission 

• Cardiac conditions and procedures (prior AMI, angina, atrial fibrillation 
[AF], hypertension, and heart failure [HF]; prior PTCA, coronary artery 
bypass graft [CABG], stent, or pacemaker,) 

• Non-cardiac condition (liver disease, diabetes, transient ischemic 
attack \J\A], stroke, chronic kidney disease [CKD], COPD, asthma, 
dyslipidemia, and tumor) 

• Other history (past/current smoking, drug use, difficulty with activities 
of daily living and other indicators of mobility limitations and frailty, 
drug allergies) 

Medication use - prior to admission and in-hospital 

• Thrombolytic use, aspirin, clopidogrel/ticlopidine, beta blockers, ACE 
inhibitors, ARBs, statins, calcium channel blockers, loop diuretic, 
thiazide diuretics, aldosterone antagonists, digoxin, amiodarone, 
warfarin, heparin, and other lipid-lowing agents 

In-hospital procedures or complications 

• Procedures - coronary angiography and findings, coronary angio- 
plasty, stent placement, echocardiography, and stress test 

• Complications - HF, hypotension, cardiogenic shock, renal failure, 
major bleed, stroke, pneumonia, sepsis, and pulmonary embolism 

In-hospital labs and test results, and results prior to discharge 

• Lab tests (as above), ECG, chest x-ray, stress test, echocardiogram, 
blood pressure 

Discharge instructions including prescribed medications. 

• Cardiovascular-related discharge medications (as above) discharge 
disposition - to home, skilled nursing, home health, etc. 



not the primary reason for the admission and may have 
been secondary to something else such as trauma, 3) 
AMI was not confirmed by ECG results, 4) AMI oc- 
curred, but patient had an invasive non-cardiac proced- 
ure within 24 hours of arrival, and 5) AMI occurred, but 
the medical record indicated that patient was to receive 
comfort or palliative care only. 

The tool was originally created in MS Excel to allow 
for ease of viewing the dimensions of care, variables, op- 
erational definitions, and to facilitate training, review, 
and revisions of the tool. 

After the tool was completed, ICE programmed the 
data elements into an electronic tool with a user-friendly 



front end interface using MS Access; this sort of tool 
had previously been used successfully by this team and 
others in similar projects [15]. Each case had a study 
identification number so that it could later be linked 
with the associated claims data files. No patient names. 
Social Security numbers, or Medicare HIC numbers 
were ever included in the database containing the ab- 
stracted data. 

Testing and fielding the data collection tool 

After pre-testing by a study team physician and the lead ab- 
stractor, the instrument was finalized and the lead ab- 
stractor trained the remaining four abstractors. Collectively, 
the abstractors had extensive medical record abstraction 
experience averaging more than ten years each; all were 
well-versed in medical terminology, though none were 
nurses or clinicians. During the training, the study objec- 
tives were described, each subsection of the tool was ex- 
plained, and instructions for each variable were highlighted. 
Each abstractor received his/her own copy of the abstrac- 
tion instruction manual and the data dictionary for 
reference while abstracting the cases. Abstractions were 
completed at a secure, onsite location exclusively. 

Prior to beginning abstraction, all abstractors initially 
reviewed the same cases, and inter-rater reliability scores 
were calculated for each data element by comparing each 
abstractors results to the "gold standard" findings from 
the lead abstractor. All discrepancies were discussed with 
the abstractors so that eventual data collected would be 
consistent across the abstractors. Feedback from this 
training exercise was used to make minor clarifications to 
the abstraction tool and the manual. When an abstractor 
demonstrated 95% agreement with the gold standard ab- 
stractor for all data elements, s/he began abstracting from 
the pool of records available. There were 32 data elements 
in the administrative and demographic portion of the tool, 
and up to 406 data elements in the rest of the tool, al- 
though the number of elements varied by case, depending 
on the health history and particular clinical management 
(tests, procedures, complications) of the patient. For ex- 
ample, if a patient had a history of stroke (a "parent" vari- 
able), the abstractor checked the "yes" box, and was 
directed to two "child" variables, regarding the severity 
and timing of the stroke. If a patient did not have a history 
of stroke, the abstractor checked the "no" box and was 
allowed to skip the next two follow-up questions. 

Medical record procurement 

The hospital claims data for the abstraction subsample 
were linked to the Provider of Services (POS) file, a pub- 
lic use file made available by CMS, to obtain hospital 
contact information for requesting the medical records. 
ICE mailed a packet to each facility with a patient in the 
study, that included a cover letter from the Principal 
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Investigator (PI, Brooks), a copy of the CMS DUA to 
document approval to request medical records, patient- 
specific identifiers (name, SSN, and HIC), and the ad- 
mission date of the hospitalization record being re- 
quested. The packet also included detailed instructions 
to photocopy the record and send all components by se- 
cure package to ICE. Hospitals were reimbursed a 
standard amount per record for photocopying and U.S. 
mail costs. 

ICE disseminated record requests for the initial sample 
(n = 1,920) in April 2012. Since an index hospitalization 
might be comprised of care at two facilities in the cases 
of a transfer, there were many instances where more 
than one record had to be obtained in order to have all 
the necessary information from the medical record. ICE 
developed a crosswalk to indicate which cases were 
comprised of a single medical record and which cases 
required a set of records. Failure to obtain both parts of 
a set would render the case un-abstractable. 

If the requested medical records from the original 
sample were not received by ICE, the medical records 
department was telephoned to re-request the record; in 
some cases, a copy of the original packet was mailed 
again. After these efforts to obtain records from the ini- 
tial sample were exhausted, record requests from the 
second wave (oversample) were disseminated on an as- 
needed basis in order to reach a quota of 25 abstracted 
records per PSU. The record request process was re- 
peated until the required number of records in each 
PSU was abstracted, or no more records were available 
to be abstracted. 

Abstraction and quality assurance 

Ongoing abstraction began in September 2012 and, ini- 
tially, aggressive weekly goals of 100 charts per week 
were established for the abstractors. If, at any time dur- 
ing the data collection process, an abstractor had a ques- 
tion or concern about how to abstract a data element, 
the on-site lead abstractor was consulted. Questions that 
required a decision from the clinical team were evalu- 
ated (by e-mail or on a weekly call) and responded to 
within approximately 48 hours. During the abstraction 
process, our study team clinicians were consulted when 
a clinical judgment was necessary, rather than requiring 
that abstractors make clinical judgments. 

To maintain quality throughout the entire project, our 
protocol involved extensive quality control for the 
abstracted records; re-abstraction of records and inter- 
rater agreement scores were obtained for 5% of the 
records abstracted. In addition to the inter-rater agree- 
ment process used for all abstractors prior to launching 
the abstraction data collection effort, ongoing internal 
quality control (IQC) processes were instituted. Two 
additional rounds of IQC were performed, both of which 



occurred after 1/3 of the cases (i.e., approximately 500 
records) were abstracted. All rounds of IQC included re- 
abstraction of a random sample of 25 cases; each of the 
abstractors completed five cases that were re-abstracted 
by the lead abstractor. An accuracy-by-variable report 
was prepared, and abstractors who did not meet the 95% 
agreement per variable standard were retrained and re- 
cords they abstracted were re-abstracted by the lead ab- 
stractor to ensure accuracy. 

To coordinate activities between all study team mem- 
bers and facilitate resolution of any barriers to abstrac- 
tion, the entire team attended biweekly meetings during 
which project updates were provided by ICE. These up- 
dates included reports detailing the number of records 
and full sets/cases requested, received, refused, and 
stopped by PSU. They also highlighted the number of 
abstracted cases that were complete to date for each 
PSU. 

Data analysis 

To ensure that data were abstracted accurately and 
uniformly across all members of the abstraction team, 
results from all three IQC rounds were aggregated 
by domain (e.g. administrative, lab values, etc. as in 
Table 2) and agreement with gold standard results was 
calculated. 

Evaluating whether nonresponse bias exists is an im- 
portant part of assessing the validity of results. Because 
of the voluntary nature of the response from the facil- 
ities, we analyzed the characteristics of patients for 
whom we received records versus those for whom we 
did not by linking patient-level beneficiary and Medicare 
A and B claims from the index AMI stay to our re- 
quested and received medical records and examined key 
dimensions that could bias results (e.g., age, gender, co- 
morbidity history, complications during the stay, length 
of stay). We also linked data from the CMS POS file to 
obtain geographic information and additional character- 
istics of the hospital facilities to allow for comparison of 
responding versus non-responding facilities. 

Results 

The project began in January 2011 when our initial data re- 
quest for Medicare enrollment and claims data was submit- 
ted to CMS for approval, and data collection ended in 
August 2013 when we met our goal of 1,600 completed 
medical record abstractions. The actual, rather than 
intended, timeline for the various study activities is depicted 
in Figure 1. Many tasks occurred simultaneously. For ex- 
ample, we did not need to have all of the claims data to 
begin developing the data collection instrument, nor did 
the instrument need to be finalized before we could begin 
requesting medical records. Each of these major activities 
took longer than we anticipated in our initial project plan. 



Cook et at. BMC Health Services Research 2014, 14:391 
http://www.bionnedcentral.conn/1472-6963/14/391 



Page 7 of 1 2 



Q2 Q3 Q4 Ql Q2 Q3 Q4 Ql Q2 













( Request & 


Create l<ey study variables; 


Examine abstracted data; formulate key derived | 






Obtain Medicare 


begin claims analysis 


variables J) 


1 




I data 


Select 


Request , 


TO 
u 






abstraction 


identifiers 






sample 


for sample J 



Mailing for 
initial 
L sample i 



Follow- 
up calls 



^ Mailing 
for2"d 
wave 



Follow- 
up calls 



Develop Tool & protocol 



Program 
Tool; pilot 
test 



( ^ 
Data 

Collection 

y ' 




^ Y 1 

Revise Data Collection 
Tool Continue Abstraction 





IQC 
#1 



IQC 

#2 



IQC 

#3 



Figure 1 Timeline of study activities. 



A total of 2,707 complete record sets were requested 
and a total of 1,765 were received for an overall response 
rate of 65%; of these, a total of 1,601 qualified for ab- 
straction. Of the complete record sets that were re- 
ceived, 164 of them involved at least one of the stop 
rules. The reasons for stopping the abstraction were, in 
order of frequency: 1) AMI was not primary reason for 
admission [120 cases], 2) no confirmation of AMI at ar- 
rival [43 cases], 3) no documented ECG within 24 hours 
of arrival [28], 4) an invasive non-cardiac procedure was 
performed within 24 hours of arrival [12 cases], and 5) 
the patient received comfort measures only [11 cases]. 
More than one stop reason may have been present. For 
91 cases, a transfer was involved and the other record 
was not received, rendering the case unusable for ab- 
straction. Figure 2 documents the medical record sample 
requested, received, and abstracted. 

To reach our goal of 1,600 abstracted records, we 
eventually relaxed our requirement that each PSU have 
exactly 25 completed cases. A total of 11 PSUs fell short 
of our goal of 25 (minimum of 21 cases) and 13 PSUs 
had more than 25 abstractions (maximum 28 cases); the 
remaining 40 PSUs had exactly 25 cases abstracted. 

The nonresponse analysis revealed that very few dif- 
ferences were found between patients whose records 
were received versus not received. Examination of 
claims-based measures at the patient-level indicated 
that patients for whom we received records tended to 
be slightly older (79 versus 78 years old at the time of 
index admission; p < 0.05), were more likely to be 



discharged home rather than to another facility (57% 
versus 52%; p < 0.0001), were less likely to require an 
acute care transfer for the AMI hospitalization (13% 
versus 21%; p < 0.0001), and had shorter average acute 
care lengths of stay (LOS) (7.2 days versus 7.9 days; p < 
0.05) (Table 3). Other demographics, as well as charac- 
teristics of the index AMI, prior comorbid conditions, 
and the complications of index stay were all 
comparable. 

An examination of the hospital facility-level non- 
response revealed that we were less likely to receive re- 
cords from facilities in the Northeast region of the U.S. 
(17% versus 26%; p < 0.0001) and from facilities in urban 
[16] areas (77% versus 81%; p < 0.05), but more likely to 
receive records from the Southern region (38% versus 
32%; p < 0.05) of the U.S (see Table 4). We were also less 
likely to receive records from larger facilities (300+ beds) 
(42% versus 47%; p < 0.05). 

The abstracted data were of high and uniform qual- 
ity. There was fairly little variation in the data between 
abstractors. The overall agreement rate for all three 
IQC samples was between 95-96%; agreement within 
each domain ranged from 82.1% (Complications and 
Conditions During the Stay) to 97% (Administrative 
and Demographic Information) (see Figure 3). Collect- 
ing some of the health history information, for example 
regarding all prior conditions and procedures, were 
more challenging for the abstractors, because legibility 
was often an issue, and some conditions were recorded 
only once, or found in unexpected places in the record. 
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Figure 2 Sampling. 
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On average, abstractors spent 45 minutes per abstrac- 
tion, ranging from a low of 30 minutes to a high of 
3 hours. 

Discussion 

Observational comparative effectiveness studies using 
electronic healthcare data are vulnerable to treatment se- 
lection bias. When supplemental data are available for a 
sample of patients, a variety of methods exist for using 
these data for sensitivity analyses. For Medicare cohorts, 
the Medicare Current Beneficiary Survey has been used in 
this way [17-19]. However, for specific research questions, 
the MCBS may be too generic to assess critical assump- 
tions and a more focused record abstraction is desirable. 
We have found that it is possible to efficiently access and 
abstract medical records and collect high quality data for 
Medicare patients at a national level to address specific re- 
search goals. 

Our high agreement level (95-96%) between abstractors 
for all data elements abstracted supports confidence in 
our data, as this rate is consistent with standards devel- 
oped for high reliability in chart review [15]. Moreover, 
since abstractors who did not meet the 95% agreement 
per variable standard were retrained and records they ab- 
stracted were re-abstracted by the lead abstractor to en- 
sure accuracy, this increases our confidence. During an 
IQC, we discovered that one abstractor scored very low; 
therefore, the lead abstractor re-abstracted all of the re- 
cords she had completed for all elements. We provided 
additional training for her and closer monitoring until we 
felt she was proficient in her abstractions. We believe that 
it is generally preferable to report kappas or intra-class cor- 
relation coefficients when quantifying inter-rater agreement 



for the abstracted data elements; however, we were not able 
to perform these types of calculations for all items due to 
concessions we made with regard to the structure of the 
data collection tool, which did not capture denominators 
for all items (e.g., grids listing all medications, check only if 
the item was present). This deliberate decision was made 
due to a limited budget for abstracting records and a very 
large amount of desired information from the records. 

The overall response rate for obtaining the requested 
records was reasonable (65%). Similar efforts at medical 
record procurement have resulted in response rates that 
range from 44%-94% (www.carecommunications.com; 
[9,20]). The beneficiary and facility level non-response 
analyses suggest that we may have experienced a slight 
bias towards receiving records for healthier patients, since 
complicated patients are often treated at or transferred to 
facilities that have a greater range of services at their dis- 
posal, are older and have longer hospitals stays -which 
result in more lengthy records for photocopying, and 
are less likely to be discharged home. Hospitals that 
responded copied an average of 1.85 records per facility; 
those that did not were asked to copy an average of 1.64; 
since hospitals were reimbursed a set amount for each 
record returned, this may have played a role in record 
procurement. Further analyses are planned to determine 
whether a weighting strategy is necessary to correct for 
these differences. 

The development of the abstraction tool took much lon- 
ger than expected, and obtaining the necessary medical re- 
cords to reach our sample quota was labor-intensive and 
time-consuming, as others have concluded [15,21]. How- 
ever, the protocol was reasonably cost efficient in that it 
could be accomplished within the budget of a small ROl 
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Table 3 Nonresponse analysis: analysis of beneficiary characteristics 




Full record received 


Full record requested but not received 


Number 


1,765 


942 


Demographics 






Percent female 


59% 


56% 


Percent dual eligible for Medicaid 


32% 


31% 


Race 






White 


83% 


84% 


Black 


6% 


7% 


Hispanic 


7% 


6% 


Other race 


3% 


4% 


Mean age 


79* 


78 


Characteristics of AMI 






Severe/Moderate/U n known 


24% 


23% 


NSTEMI (not severe) 


76% 


76% 


Comorbid conditions 






Unstable angina 


30% 


33% 


Heart Failure 


48% 


47% 


Hypertension 


73% 


77% 


Complications of index stay 






Cardiac arrest 


3% 


2% 


Ventricular arrhythmia 


8% 


8% 


Other cardiac arrhythmia 


50% 


52% 


Atrial Fibrillation 


21% 


19% 


Stroke 


3% 


2% 


Characteristics of hospital stay 






Multiple acute care facility/transfer 


13%*** 


21% 


Discharge disposition 






Discharged home 


57%*** 


52% 


Discharged to HH/another facility 


43%*** 


48% 


Mean LOS 


7.2* 


7.9 



*p < .05; *^p,.001; *^^p < .0001. 



grant (total grant budget $1.07 M over four years). Some 
hospitals refused to send records on the grounds that they 
first required the patients permission prior to releasing 
medical information; in addition, it is possible that a more 
rapid cycle of follow-up reminders than was used for this 
study might improve results, as many facilities could not 
recall receiving a medical record request for our study by 
the time we made follow-up calls - sometimes several 
months later. This resulted in having to re-send an entire 
bulky packet with letters and DUAs; whereas possibly a 
post card or letter reminder to non-respondents earlier in 
the process might have been less expensive and potentially 
resulted in earlier receipt of records. A total of 77 of the 
cases (about 5%) were abstracted based on PDF files that 
were created from electronic health records (EHRs). Our 



abstractors noted that the files that were indexed, such as 
labs and physician orders, took less time to abstract, be- 
cause it was easier to find to the correct section and locate 
the necessary information. Non-indexed PDF files from 
EHRs did not have this advantage, and took as long as 
paper records to abstract because one had to navigate 
each page at a time to find the information. Overall, there- 
fore, the contribution of EHRs to our project progress was 
minimal, although it was probably easier for hospitals that 
kept EHRs to respond to our request. 

With regard to "stopped" cases, we are aware that for 
our project, not having a confirmed AMI is a potential 
confounder when analyzing the effectiveness of pharma- 
cotherapy regimens using Medicare claims data. We 
plan on assessing the distribution of "stopped" patients 
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Table 4 Nonresponse analysis: analysis of facility level characteristics 



Facility characteristics 





Facility returned one or more records 


Facility did not return any records 


Number (%) 


939 (82%) 


200 (18%) 


Hospital Type 






For profit 


13% 


13% 


NFP 


87% 


87% 


Number of liospital beds 






Under 100 


16% 


14% 


100-199 


24% 


22% 


200-299 


18% 


17% 


300+ 


42%* 


47% 


Region 






Nortlieast 


1 7%*** 


26% 


Midwest 


26% 


24% 


South 


38%* 


32% 


West 


18% 


18% 


Metropolitan Status 






Urban/Metro 


77%* 


81% 


Nonmetro 


23%* 


19% 



*p < .05; < .001.***p<.0001. 



between treated and non-treated groups, and between 
groups based on area treatment rates, to determine 
whether analytical adjustments will be required 

Our next step is to address the primary objective of 
evaluating model assumptions and statistically bounding 
our claims-based estimates of treatment effectiveness. 
We will estimate the mean differences in each of the 



construct measures between the patients receiving each 
of the eight treatment combinations, evaluating the stat- 
istical significance of any differences using analysis-of- 
variance methods (ANOVA). 

We are in the process of developing algorithms to assess 
the severity of AMI, the presence and severity of comorbidi- 
ties at admission, contraindications to treatment, functional 



m 



.5% 95.4% 




Figure 3 IQC by domain. 
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status, the burden of atherosclerosis, and complications dur- 
ing the AMI hospitalization. These algorithms will be tested 
and validated by comparing to existing approaches in the lit- 
erature prior to finalization. These measures will serve as a 
foundation for the next stage of analysis, which will focus on 
bounding both RA and IV estimates using chart abstraction 
data to identify unmeasured confounders. For RA estimates, 
the chart abstraction data will allow us to compare the dis- 
tributions of unmeasured confounding variables between 
treated and untreated patients. 

These measures will also assist us in the interpretation 
of our IV estimates. The IV estimator in our study group 
patients based on ATRs around their residences and ex- 
ploits the treatment rate differences across these patient 
groups to yield treatment effect estimates. The under- 
lying assumption in this approach is that unmeasured 
confounders are evenly distributed across patients 
grouped by ATRs. To assess the validity of this assump- 
tion, we will group the chart abstraction patients based 
on their local ATRs and compare measures from the 
chart abstraction between groups. For example, when 
evaluating the assumptions underlying the incremental 
benefit of a statin for the patients receiving a statin only, 
we will compare patients living in areas with high statin- 
only utilization rates with patients living in areas with 
high "no treatment" utilization rates. We will use these 
comparisons to describe potential violations in IV as- 
sumptions and the bias directions in our LATE estimates 
resulting from these violations, and potentially help us 
"bound" the true LATE effects. 

Finally, these data were collected to address specific 
grant objectives related to unmeasured confounders by 
our instrument and the treatment received by the pa- 
tient, but we believe these medical record data will be 
helpful in addressing other research questions, such as 
adherence to guideline medications. However, since the 
patients for whom we obtained data were not randomly 
selected for abstraction, we plan to use the Medicare 
data that was used to generate our original abstraction 
sample to calculate and apply appropriate weights for 
each abstracted case. Weighting variables will include 
those used for sampling (ATR values, treatment combi- 
nations and geographic region). We will also determine 
whether any patient demographic, index hospital charac- 
teristics, or treatment factors (such as being transferred 
to another hospital) should be included in the weighting 
equation. As a result, we should be able to address a 
wide variety of research questions related to AMI pa- 
tients and the care they receive during and after their 
hospital stay. 

Conclusion 

Studies using observational data, even when researchers 
employ instrumental variable methods, are almost always 



subject to criticism regarding the possibility of treatment 
selection bias. We believe our methods for collecting, 
abstracting, and interpreting the important clinical infor- 
mation contained in medical records from Medicare pa- 
tients provide a valuable approach for researchers to 
address this issue; not only it is possible to obtain medical 
records from hospitals on a voluntary basis, but also to 
collect high quality clinical data to more thoroughly evalu- 
ate the effectiveness and appropriateness of care. 
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